smartrac1 smartrac2 223.223.80.71 223.223.80.72 16.16.16.1 16.16.16.2


1.VBox5.0 安装和下载（略） 2. 先各自安装RHEL6.8，进入系统后关机。 3.创建一个磁盘为固定大小磁盘，设置为共享属性，附加到2个虚机中即可。 4. 再启动两个虚机客户端。
VMware 的模拟硬件环境搭建:(类似，只是共享盘需要用命令) C:Program Files (x86)VMwareVMware Workstation>vmware-vdiskmanager.exe -c -s 10240M -a lslogic -t 2 "C:ShareDisksharedisk-1.vmdk" C:Program Files (x86)VMwareVMware Workstation>vmware-vdiskmanager.exe -c -s 10240M -a lslogic -t 2 "C:ShareDisksharedisk-2.vmdk"
C:Program Files (x86)VMwareVMware Workstation>vmware-vdiskmanager.exe -c -s 10240M -a lslogic -t 2 "C:ShareDisksharedisk-3.vmdk"
C:Program Files (x86)VMwareVMware Workstation>vmware-vdiskmanager.exe -c -s 10240M -a lslogic -t 2 "C:ShareDisksharedisk-4.vmdk"
更改VMX文件。 disk.locking="FALSE" scsi0:1.SharedBus="Virtual" scsi0:2.SharedBus="Virtual" scsi0:3.SharedBus="Virtual" scsi0:4.SharedBus="Virtual" disk.EnableUUID = "TRUE"

5. 配置HOST: [root@rac1 etc]# vim /etc/hosts #127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 rac1.sdsmart.cn rac1 ---一定要注释掉 #::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 rac1.sdsmart.cn rac1---一定要注释掉 127.0.0.1 localhost ::1 localhost # Public 223.223.83.110 rac1 223.223.83.112 rac2 # Private 16.16.16.1 rac1-pri 16.16.16.2 rac2-pri # Virtual 223.223.83.90 rac1-vip 223.223.83.92 rac2-vip # SCAN 223.223.83.99 rac-scan
6.配置sysctl.conf (固定不变的) vim /etc/sysctl.conf fs.aio-max-nr = 1048576 fs.file-max = 6815744 kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 net.ipv4.ip_local_port_range = 9000 65500 net.core.rmem_default=262144 net.core.rmem_max=4194304 net.core.wmem_default=262144 net.core.wmem_max=1048586 执行 #/sbin/sysctl –p
7.修改/etc/security/limits.conf vim /etc/security/limits.conf grid soft nproc 2047 grid hard nproc 16384 grid soft nofile 1024 grid hard nofile 65536 grid soft stack 10240 grid hard stack 32768 oracle soft nproc 2047 oracle hard nproc 16384 oracle soft nofile 1024 oracle hard nofile 65536 oracle soft stack 10240 oracle hard stack 32768
8.修改/etc/pam.d/login vim /etc/pam.d/login session required pam_limits.so
9.禁止网络时间服务、防火墙 service ntpd stop chkconfig ntpd off
service iptables stop chkconfig iptables off
10.关闭SELinux vi /etc/selinux/config SELINUX=disabled


11. 创建用户、组和文件路径 groupadd -g 1100 oinstall
groupadd -g 1200 asmadmin
groupadd -g 1101 dba
groupadd -g 1102 oper
groupadd -g 1201 asmdba
groupadd -g 1202 asmoper
useradd -u 1100 -g oinstall -G dba,oper,asmdba -d /home/oracle -m oracle

useradd -u 1200 -g oinstall -G asmadmin,asmdba,asmoper,dba -d /home/grid -m grid

--rm /var/spool/mail/oracle --userdel -r oracle
useradd -u 1100 -g oinstall -G dba,oper,asmdba -d /home/oracle -m oracle usermod -u 1100 -g oinstall -G dba,oper,asmdba -d /home/oracle -m oracle
useradd -u 1200 -g oinstall -G asmadmin,asmdba,asmoper,dba -d /home/grid -m grid usermod -u 1200 -g oinstall -G asmadmin,asmdba,asmoper,dba -d /home/grid -m grid
12.创建目录授权 mkdir -p /u01/app/grid mkdir -p /u01/app/11.2.0/grid mkdir -p /u01/app/oracle/product/11.2.0/db_1 mkdir -p /orasoft/ora11g
--更改属主,权限 chown -R grid:oinstall /u01/app chown -R oracle:oinstall /u01/app/oracle chown -R grid:oinstall /orasoft/ora11g chmod -R 777 /orasoft/ora11g
13.RPM包: cd /media/RHEL-6.8 Server.x86_64/Packages rpm -Uvh binutils-2.* rpm -Uvh compat-libstdc++-33* rpm -Uvh elfutils-libelf-0.* rpm -Uvh elfutils-libelf-devel-* rpm -Uvh gcc-4.* rpm -Uvh gcc-c++-4.* rpm -Uvh glibc-2.* rpm -Uvh glibc-common-2.* rpm -Uvh glibc-devel-2.* rpm -Uvh glibc-headers-2.* rpm -Uvh ksh-2* rpm -Uvh libaio-0.* rpm -Uvh libaio-devel-0.* rpm -Uvh libgcc-4.* rpm -Uvh libstdc++-4.* rpm -Uvh libstdc++-devel-4.* rpm -Uvh make-3.* rpm -Uvh sysstat-7.* rpm -Uvh unixODBC-2.* rpm -Uvh unixODBC-devel-2.*




14.搭建YUM本地仓库 [root@rac1 mnt]# cp -rf /media/RHEL-6.8 Server.x86_64/ /mnt/ [root@rac1 mnt]# mv /mnt/RHEL-6.8 Server.x86_64 /mnt/
cd /etc/yum.repos.d/ vim vim rhel-source.repo name=Red Hat Enterprise Linux $releasever - $basearch - Source #baseurl=ftp://ftp.redhat.com/pub/redhat/linux/enterprise/$releasever/en/os/SRPM S/ baseurl=file:///mnt/redhat/ enabled=1 gpgcheck=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
保存后，执行 yum makecache yum install binutils yum install compat-libstdc++-33* yum install elfutils-libelf-0.* yum install elfutils-libelf-devel-* yum install gcc-4.* yum install gcc-c++-4.* yum install glibc-2.* yum install glibc-common-2.* yum install glibc-devel-2.* yum install glibc-headers-2.* yum install ksh-2* yum install libaio-0.* yum install libaio-devel-0.* yum install libgcc-4.* yum install libstdc++-4.* yum install libstdc++-devel-4.* yum install make-3.* yum install sysstat-7.* yum install unixODBC-2.* yum install unixODBC-devel-2.*


15.建立用户环境变量 --grid 用户 ORACLE_BASE=/u01/app/grid; export ORACLE_BASE ORACLE_HOME=/u01/app/11.2.0/grid; export ORACLE_HOME GRID_HOME=/u01/app/11.2.0/grid; export GRID_HOME ORACLE_SID=+ASM2; export ORACLE_SID TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN PATH=${PATH}:$ORACLE_HOME/bin:$ORACLE_HOME/OPatch; export PATH LD_LIBRARY_PATH=$ORACLE_HOME/lib LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib export LD_LIBRARY_PATH CLASSPATH=$ORACLE_HOME/JRE CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib export CLASSPATH THREADS_FLAG=native; export THREADS_FLAG export TEMP=/tmp export TMPDIR=/tmp export NLS_DATE_FORMAT="YYYY-MM-DD HH24:MI:SS"
if [ $USER = "oracle" ] || [ $USER = "grid" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi fi
--oracle 用户 红字部分（rac1 改为 rac1的数据库实例名,rac2 改为 rac2 的数据库实例名) ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1; export ORACLE_HOME ORACLE_SID=gnnt2; export ORACLE_SID TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN PATH=${PATH}:$ORACLE_HOME/bin:$ORACLE_HOME/OPatch; export PATH LD_LIBRARY_PATH=$ORACLE_HOME/lib LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib export LD_LIBRARY_PATH CLASSPATH=$ORACLE_HOME/JRE CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib export CLASSPATH THREADS_FLAG=native; export THREADS_FLAG export TEMP=/tmp export TMPDIR=/tmp export NLS_LANG=AMERICAN_AMERICA.AL32UTF8 export NLS_DATE_FORMAT="YYYY-MM-DD HH24:MI:SS"
if [ $USER = "oracle" ] || [ $USER = "grid" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi fi
16.建立SSH认证(oracle/grid) 非常重要 vi sshx (在oracle/grid下均建立此脚本) mkdir -p ~/.ssh chmod -R 755 ~/.ssh /usr/bin/ssh-keygen -t rsa /usr/bin/ssh-keygen -t dsa cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys echo -n "wait another host $1 run finish, press twice key continue !!!" read a read a ssh $1 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys ssh $1 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys chmod -R 644 ~/.ssh/authorized_keys
--授权 chmod 755 ~/sshx
建立完毕后互相测试是否不需要密码认证 非常重要 ssh rac1.sdsmart.cn date ssh rac2.sdsmart.cndate ssh rac1 date ssh rac2 date ssh rac1-pri.sdsmart.cn date ssh rac2-pri.sdsmart.cn date ssh rac1-pub.sdsmart.cn date ssh rac2-pub.sdsmart.cn date
17.设置时间同步 vi /etc/ntp.conf server 10.135.8.100 prefer 也可以本地设定日期时间： $ date -s '2016-07-11 16:55:00'

如果仅需要设置日期或者时间，可以使用date -d mm/dd/yy及date -s hh:mm:ss。

18.创建UDEV --- 创建个文件 echo "options=--whitelisted --replace-whitespace" >> /etc/scsi_id.config --- 添加RULE --5块硬盘(/dev/sdb sdc sdd sde sdf) for i in b c d e ; do echo "KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="`/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/sd$i`", NAME="asm-disk$i", OWNER="grid", GROUP="asmadmin", MODE="0660"" >> /etc/udev/rules.d/99-oracle-asmdevices.rules done
查看结果如下： cat /etc/udev/rules.d/99-oracle-asmdevices.rules rm /etc/udev/rules.d/99-oracle-asmdevices.rules --检查是否可写 dd if=/dev/sdb of=/dev/null bs=1M count=1

19. 启动UDEV /sbin/start_udev
[root@rac1 rules.d]# cat 99-oracle-asmdevices.rules KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB1faf0bb4-32ab5fcf", NAME="asm-diskb", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB1cab8d6b-307a9107", NAME="asm-diskc", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VBac63a170-a1d9ad9d", NAME="asm-diskd", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VBd1c2dd31-a66738c5", NAME="asm-diske", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB445d4c5e-9312d88b", NAME="asm-diskf", OWNER="grid", GROUP="asmadmin", MODE="0660" [root@rac1 rules.d]# ls -l /dev/asm* ls: cannot access /dev/asm*: No such file or directory [root@rac1 rules.d]# [root@rac1 rules.d]# [root@rac1 rules.d]# [root@rac1 rules.d]# start_udev Starting udev: [ OK ] [root@rac1 rules.d]# ls -l /dev/asm* brw-rw---- 1 grid asmadmin 8, 16 Jul 7 10:27 /dev/asm-diskb brw-rw---- 1 grid asmadmin 8, 32 Jul 7 10:27 /dev/asm-diskc brw-rw---- 1 grid asmadmin 8, 48 Jul 7 10:27 /dev/asm-diskd brw-rw---- 1 grid asmadmin 8, 64 Jul 7 10:27 /dev/asm-diske brw-rw---- 1 grid asmadmin 8, 80 Jul 7 10:27 /dev/asm-diskf [root@rac1 rules.d]#
20.rz传输压缩包，解压缩 ### grid 用户 unzip p13390677_112040_Linux-x86-64_3of7.zip ### oracle用户 unzip p13390677_112040_Linux-x86-64_1of7.zip unzip p13390677_112040_Linux-x86-64_2of7.zip unzip p19121551_112040_Linux_x86_64.zip 检查是否正常，可进行安装 [grid@rac1 ~]$ cd /mnt/redhat/oracle/grid/ [grid@rac1 grid]$ ./runcluvfy.sh stage -pre crsinst -n smartrac1,smartrac2 -verbose > /tmp/runcvuvfy.out 给出的报告可以忽略掉 pdksh 和 elfunits 两个组件包，也可自行安装. elfutils-libelf-devel-0.97 pdksh-5.2.14
21 安装、配置Grid的三方RPM包 cd /mnt/redhat/oracle/grid/rpm rpm -ivh cvuqdisk-1.0.9-1.rpm CVUQDISK_GRP=oinstall; export CVUQDISK_GRP

22.安装GRID 需要进到主机进行图形化安装，分辨率设置再1024以上。

最后执行脚本 rac1 rac2 上执行脚本:/u01/app/oraInventory/orainstRoot.sh rac1 rac2 上执行脚本:/u01/app/11.2.0/grid/root.sh
23.进行asmca配置 grid用户执行 asmca命令。 增加两个区: FRA 是 快速恢复区，DATA是放数据文件的 需要了解的 NORMAL:镜像 EXTERNAL: 独立硬盘设备。
24.安装ORACLE服务器端程序。 1.进入Oracle用户 2.图形化安装。

暂时忽略以上错误，点击下一步。





24. RAC关机顺序指令: --- rac1

su - oracle
sqlplus / as sysdba
shutdown immediate

su - grid
alter diskgroup data dismount;
alter diskgroup fra dismount;


--- rac2

su - grid
alter diskgroup data mount;
alter diskgroup fra mount;

上述这个步骤主要就是让oracle 进行 ASM INSTANCE RECOVERY

su - oracle
sqlplus / as sysdba
startup


--- rac1
su - grid
alter diskgroup data mount;
alter diskgroup fra mount;

su - oracle
sqlplus / as sysdba
startup




异常问题处理 --卸了重装 ./runInstaller -silent -deinstall -removeallfiles "REMOVE_HOMES={/u01/app/oracle/product/11.2.0_1}" -invPtrLoc /etc/oraInst.loc
--检查集群状态
crsctl check cluster --所有Oracle实例（数据库状态） srvctl status database -d YMS --节点应用程序状态 status nodeapps --列出所有的配置数据库 srvctl config database --数据库配置 srvctl config database -d YMS -a --手工注册监听 alter system register;
--查看RAC节点状态 crsctl stat res -t
--查看节点 crs_stat -t -v
--启动单实例 srvctl enable instance -d ractest -i ractest1
srvctl relocate scan_listener -i 1 -n rac2
--启动监听 srvctl start listener -n rac1
-- 日志路径
集群日志: /u01/app/11.2.0/grid/log
ASM INSTANCE日志： /u01/app/grid/diag/asm/+asm
DB INSTANCE日志： /u01/app/oracle/diag
//拷贝迁移虚拟机造成IP网卡无法在正常运行的方法。 1.先用ifconfig看看当前网卡的实际物理地址 2.更改 /etc/sysconfig/network-scripting 下的 ifcfg-ethXX 拷贝ifconfig看到的物理地址到 ifcfg-ethXX 3.更改/etc/udev/rules.d/70-persistent-net.rules对应的物理地址即可。
增加虚拟内存: